Apparatus for presenting support images to a driver and method thereof

ABSTRACT

An image recognition unit recognizes an object in each image captured by a plurality of imaging devices each imaging a partial overlapped imaging region surrounding a vehicle. An image synthesis unit generates a synthesized image by partially overlapping and joining converted images viewed from preset virtual viewpoints, when an imaging region including the recognized object is an overlapping region of which any one of two imaging devices performs imaging. The image synthesis unit synthesizes each converted image area in the overlapping image regions, using a preset blend ratio when an image of the converted images which are joined to partially overlap is generated. An image-setting unit sets the blend ratio of each converted image in the overlapping image regions for recognition results of an object, and a driving support image synthesized with the set blend ratio is output to a display apparatus.

TECHNICAL FIELD

The present invention relates to display control technique for providingdriving support images to a driver of a vehicle, and more particularly,relates to a display control technique which provides driving supportimages to a driver, based on image data obtained from a plurality ofimaging devices, each imaging device capturing images in each regionwhich overlaps in part with another region surrounding the vehicle inwhich the imaging devices are mounted.

RELATED ART

Conventionally there is a known technique of controlling on vehicledisplay, by imaging each imaging region, which is partially overlappedsurrounding a vehicle, using a plurality of cameras mounted on thevehicle. Joining converted images of the plurality of images capturedgenerates a single display image, and the display image is output to theon-vehicle display as a driving support image.

For example, after the plurality of captured images is converted into abird's eye view image by joining the images, a single bird's eye viewdisplay image is generated. At this point, there are however cases of anobject being detected in an imaging region corresponding to the joinedsection of the bird's eye view display image. In such a case, atechnique described in JP2007-41791A proposes a technique of changing aposition of an image boundary line which is the joining section of thebird's eye display image.

CITATION LIST Patent Literature

[Patent Literature 1] JP2007-41791A

Technical Problem

However, according to conventional techniques, by displaying the joinedsection of the image as the boundary line in the driving support image,a problem of displaying unnatural, for example, distorted drivingsupport images to a driver may arise.

In this regard, it is considered that the problem of such unnaturalappearance of the display image may be resolved by joining two convertedimages to synthesize an overlapping image region. However, when anobject exists in an overlapping image area of a synthesized image,deterioration of visibility of the object is also considered to bepotentially troublesome for the driver.

SUMMARY

In view of the above issues, the present disclosure aims to provide adisplay control technique which resolves unnatural appearance of joinedsections of a displayed image, by synthesis of image region areas whichoverlap each converted image, and suppresses deterioration of thevisibility of an object in the overlapping image region areas.

An aspect of the disclosure is a display control apparatus installed ina vehicle. The display control apparatus is provided with an imagerecognition unit, an image synthesis unit, an image-setting unit and animage-outputting unit. The image recognition unit recognizespredetermined objects from images captured by a plurality of imagingdevices mounted on the vehicle in order to capture images in eachpartially overlapped imaging region surrounding the vehicle.

An imaging region which has an object recognized by the imagingrecognition unit included in the region may be an overlapping region inwhich both of two imaging devices are operable to capture images. Inthis case, the image synthesis unit converts each captured image of thetwo imaging devices to images which are each viewed from a presetvirtual viewpoint, and generates an adjoined synthesized image bypartial overlapping of each converted image. The image synthesis unitsynthesizes each converted image area in the overlapping image regionsusing a preset blend ratio, when the synthetic image is synthesized byjoining each of the converted images to provide partially overlappedsections. That is, the overlapping image regions are regions in whicheach converted image is overlapped to fit with another converted image.

The image-setting unit sets the blend ratio of each converted imageareas in the overlapping image regions based on recognition results fromthe image recognition unit. In this way, each of the converted imagesareas in the overlapping image regions is synthesized by the imagesynthesis unit using a blend ratio set by the image-setting unit. Thesynthesized image is output from the image-outputting unit to a displayapparatus installed in the vehicle as a driving support image.

According to this configuration, the converted image areas in theoverlapping image regions are synthesized by joining to partiallyoverlap each of the converted images and the converted image areas aresynthesized using a set blend ratio based on the recognition results ofan object. Furthermore, an image may be presented without enhancing thejoined sections and clearer presentation of an object in overlappingimage region areas, which include a joined section, may be thusachieved.

According to the present disclosure, unnatural appearance, such asdistortion of a joined section is resolved by synthesis of theoverlapping image region areas, and visual deterioration of an object inthe overlapping image region areas may also be suppressed.

Also, the same effect as the display control apparatus mounted in avehicle as the foregoing aspect of the disclosure may be also obtainedwith a display control method according to another aspect of thedisclosure, for the same reasons mentioned hereinabove.

It is to be understood that symbols in the summary and claims are usedto show a corresponding relation between specific means as a modedescribed in preferred embodiments, and do not limit a technical rangeof the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

In the accompanying drawings:

FIG. 1 is a block diagram showing a configuration of a display controlapparatus 1 mounted in a vehicle according to an embodiment;

FIG. 2 is a descriptive drawing showing each imaging region of aplurality of cameras 10 mounted on a host vehicle according to theembodiment;

FIG. 3 is a block diagram showing a functional configuration of adisplay control unit 20 according to the embodiment;

FIG. 4 is a descriptive diagram which includes a basic correspondencetable (in FIG. (A)) and a supplementary correspondence table (in FIG.(B));

FIG. 5 is a descriptive diagram which includes description of a virtualviewpoint in a front and rear direction of the host vehicle (in e FIG.(A)), a description of the virtual viewpoint in a left-right directionof the host vehicle (in FIG. (B)), and a descriptive diagram of thevirtual viewpoint in an oblique direction of the host vehicle (same FIG.(C));

FIG. 6 includes an image showing a synthesized image (in FIG. (A)), andan image showing a driving support image with an enhanced image addedthereto;

FIG. 7 is a flow chart showing a virtual viewpoint setting process ofthe embodiment;

FIG. 8 is a flowchart showing a simplified table selection processaccording to the embodiment;

FIG. 9 is a flowchart exemplifying a blend ratio setting processaccording to the embodiment; and

FIG. 10 is a descriptive diagram including an image of a viewpointconverted image area of an object based on captured image from a rearcamera 4 (in FIG. (A)), an exemplified image of the viewpoint convertedimage area of the object based on the captured image of a right-sidecamera 6 (in FIG. (B)), an exemplified image of an overlapping imagearea of both viewpoint converted image areas synthesized at a 50% blendratio (in FIG. (C)), and an exemplified image of the overlapping imagearea of the both viewpoint converted image areas synthesized at a blendratio of 70%:30% (in FIG. (D)).

EMBODIMENTS OF DISCLOSURE

Embodiments of the present disclosure will now be described withreference to figures.

First Embodiment

A display control apparatus 1 mounted in a vehicle is provided with aplurality of cameras 10, a display control unit 20 and a display 30.Although not shown in the figures, the display control apparatus 1mounted in a vehicle is connected to an in-vehicle Local Area Net Work(referred to as an in-vehicle LAN hereon). The LAN is configured toshare vehicle related information, for example, information detectedfrom each type of sensor between other Electronic Control Units(referred to as ECU hereon) which are connected to the in-vehicle LAN.

A vehicle in which configuring elements are mounted is referred to as ahost vehicle. It is to be understood that the display control unit 20 inthe first embodiment is a processing device which processes image datacaptured by the plurality of cameras 10 (described in detailhereinafter).

It is noted that the in-vehicle LAN is a Local Area Network deployedinside the host vehicle. The LAN uses a communication protocol, such as,CAN (Controller Area Network), FlexRay, LIN (Local Interconnect NetWork), MOST (Motor Oriented Systems Transport Network), ACV-LAN (Audioand Video Communication LAN), for example, to transmit each type ofvehicle related information. In the first embodiment, informationshowing a driving direction of the host vehicle (for example,shift-lever position, steering direction, and other manipulatedvariables, such as, an accelerator pedal variable) are transmitted fromother ECU to the display control apparatus 1, mounted in the vehicle, asvehicle related information.

The cameras 10 are a plurality of imaging devices mounted on the hostvehicle used for imaging the surrounding of the host vehicle. Morespecifically, each camera 10 is mounted on a respective front, rear,left, and right position of the host vehicle. In the first embodiment,each camera 10 is broadly classified as a front camera 2, a rear camera4, a right-side camera 6 and a left-side camera 8, according to themounted position and imaging region of each camera mounted on the hostvehicle. As shown in FIG. 2, the front camera 2 is mounted on a frontsection of the host vehicle (for example, in a center of the frontsection) and images a front region A1 of the host vehicle. It is notedthat the rear camera 4 is mounted on a rear section of the host vehicle(for example, in a center of the rear section) and images a rear regionA2 of the host vehicle. The right-side camera 6 is mounted on aright-side section of the host vehicle (for example, a right-side backmirror-section) and images a right-side region A3 of the host vehicle.The left-side camera 8 is mounted on a left-side section of the hostvehicle (for example, a left side back mirror section) and images aleft-side region A4 of the host vehicle.

Each camera 10 is mounted on the host vehicle so that a part of eachimaging region is overlapped with a part of an imaging region of atleast one different camera 10 (referred to as overlapping region hereonin the specification of the present disclosure). For example, as shownin FIG. 2, the front region A1 which is the imaging region of the frontcamera 2, includes a front-right overlapping region OA1 and a front-leftoverlapping region OA2. The front-right overlapping region OA1specifically overlaps with a part of the right-side region A3 which isthe imaging region of the right-side camera 6, and the front-leftoverlapping region OA2 specifically overlaps with a part of theleft-side region A4 which is the imaging region of the left-side camera8. In the same way, the rear region A2 which is the imaging region ofthe rear camera 4, includes a rear-right overlapping region OA3 and arear-left overlapping area OA4. The rear-right overlapping region OA3specifically overlaps with a part of the right-side region A3 which isan imaging region of the right-side camera A6, and the rear-leftoverlapping area OA4 specifically overlaps with a part of the left sideregion A4 which is an imaging area of the left-side camera 8. That is,the overlapping regions are regions in which any one of the two cameras10 is operable to capture images.

As shown in FIG. 2, in the front region A1, a region other than thefront-right overlapping region OA1 and the front-left overlapping regionOA2 is referred to as a front single region SA1. In the rear region A2,a region other than the rear-right overlapping region OA3 and therear-left overlapping region OA4 is referred to as a rear single regionSA2. In the same manner, in the right-side region A3, a region otherthan the front-right overlapping region OA1 and the rear-rightoverlapping region OA3 is referred to as a right-side single region SA3.In the left side region A4, a region other than the front-leftoverlapping region OA2 and the rear-left overlapping region OA4 isreferred to as a left-side single region SA4. That is, imaging in thefront region A1 which can be performed only by the front camera 2 isreferred to as the front single region SA1, and imaging in the rearregion A2 which can be performed only by the rear camera is referred toas the rear single region SA2. Furthermore, imaging on the right-sideregion A3 which can be performed only by the right-side camera 6 isreferred to as the right-side single region SA3, and imaging on theleft-side region A4 which can be performed only by the left-side camera8 is referred to as the left side single region SA4.

The display 30 is installed in the host vehicle, for example, as adisplay apparatus. For example, the display 30 is configured of a liquidcrystal display, a head-up display or both of the mentioned displayscombined, and mounted in a position which is easily seen by the driverof the host vehicle.

The display control unit 20, that is, a processing device is mainlyconfigured of a known micro-computer and an in-vehicle LAN communicationcontroller. The micro-computer is provided with a CPU 12 (CentralProcessing Unit), and a RAM 14A (Random Access Memory), a ROM 14B (ReadOnly Memory) and a semi-conductor memory 14C such as a flash memory. Therespective RAM 14A, ROM 14B and semiconductor memory 14C will be simplyreferred to as ‘memory 14’ hereon. The CPU 12 executes each type ofprocess, on the basis of a digital computer program stored in the memory14. Specifically, this program executes a method corresponding to adisplay control program, for example.

It is to be understood that one micro-computer or a plurality ofmicro-computers may be provided in the display control unit 20, eachmounting place of the single or plurality of micro-computers may beprovided anywhere inside the host vehicle. It is also noted that the ROM14B of the memory 14 functions as a non-transitory storage media.

The display control unit 20 is functionally provided with an imagerecognition unit 21, a viewpoint conversion unit 22, an image synthesisunit 23, an image-setting unit 24, an image enhancement unit 25 and animage-outputting unit 26, which is functionally actualized by executionof each type of process performed by the CPU 12. The display controlunit 20 may also be configured hardware to execute these functions inpart or entirely using one or a plurality of electronic circuits, forexample, logic circuits or IC circuits.

The image recognition unit 21 is provided with a function to recognizepredetermined objects in each image. The object is a pedestrian oranother vehicle, for example. It is considered that the driver of thehost vehicle is desirably informed of the presence of the object, inview of driving support. Recognition of an object includes a process inwhich calculation of a candidate value indicating a probability of theobject as a candidate object, (a value which indicates a recognitionprobability of the object) and a movement value indicating the speed ofthe candidate object, for example, is performed by detection and tracingthe candidate object.

The detection of the candidate object, for example, entails detection ofa part of an object, which satisfies a featured element predeterminedfor each object in an image. A featured quantity indicating aqualitative and quantitative level of the featured element to a degreeof the detected candidate object is stored in the memory 14. Tracing ofthe candidate object is performed, for example, by using a plurality ofimages continuously captured along a time series, to obtain an opticalflow value illustrated as a vector of movement of the candidate objectin the continuously captured images, whereby the optical flow value isstored as the movement value of the candidate object in the memory 14.Specifically, the candidate object is recognized as the object whenpredetermined conditions based on the information are satisfied. It isnoted that, a recognition method of an object is a method known toperson in the field, therefore a detailed explanation is omitted.

Identifying information which identifies a captured image which includesa recognized object, from other captured images, and information of animage position which indicates a position related to the recognizedobject in an image, for example, are stored in the memory 14. It isnoted that the image position information includes informationspecifying any one of the single regions SA1 to SA4 and overlappingregions OA1 to OA4 as an imaging region, which includes an objectrecognized by the image recognition unit.

The viewpoint conversion unit 22 specifies one or a plurality of cameras10 which capture images having a recognized object, recognized by theimage recognition unit, as an image recognition device (also referred toas recognition camera hereon). The viewpoint conversion unit 22 isprovided with a function of converting a captured image to a viewpointimage, viewed from a virtual viewpoint, which is pre-designated for therecognition camera. It is noted that a process which actualizes afunction of specification of the recognition camera, and setting thevirtual viewpoint, will be described herein after. The viewpointconversion unit 22 according to the first embodiment, converts capturesimages from all of the cameras 10 as candidate viewpoint convertedimages, and each viewpoint converted image is then supplied to the imagesynthesis unit 23.

The viewpoint converted image may also be referred to as a coordinateconversion image. The coordinate conversion image is a captured imageviewed from a viewpoint of the camera 10 which has coordinates convertedto the virtual camera viewpoint. For example, if an optical axis of acamera coordinate is taken as a reference, coordinate positions of allpoints on an image is calculated by an angle and distance of the pointsfrom the optical axis, hence the viewpoint conversion of the image maybe performed by rotation and translation of the coordinate positions onthe basis of the optical axis of the virtual camera. Specifically, ifthe position and direction of the virtual viewpoint is set as theoptical axis of the virtual camera, a desired viewpoint converted imagemay be obtained.

It is noted, since viewpoint conversion techniques of images are knowntechniques a detailed description is omitted.

A correspondence table, which is pre-stored in the memory 14, is used toset a position and direction of the virtual viewpoints. This table isdivided into a basic correspondence table and a supplementarycorrespondence table. The basic correspondence table provides eachcamera 10 and the relative virtual viewpoint in a one to onecorrespondence, as exemplified in FIG. 4(A). The basic correspondencetable thus enables unambiguous setting of the virtual viewpoint when oneof the cameras 10 is specified as the recognition camera.

In contrast, as shown in FIG. 4(B), the supplementary table provides oneto one correspondence of each combination of the cameras 10 and avirtual viewpoint. For example, the cameras 10 have four combinationpatterns which include; a combination of the front camera 2 and theright-side camera 6 (refer to FIG. 5(C), virtual viewpoint E5), acombination of the front camera 2 and the left-side camera 8 (refer toFIG. 5 (C), virtual viewpoint E6), a combination of the rear camera 4and the right-side camera 6 (refer to FIG. 5 (C), and virtual viewpointE7), and a combination of the rear camera 4 and the left-side camera 8,(refer to FIG. 5 (C) and virtual viewpoint E8). That is, thesupplementary table enables unambiguous setting of the virtual viewpointfor the specified camera 10 as the recognition camera in any one of thecombination patterns mentioned above. It is noted that each one of thefour combinations is the same as the two cameras 10 used to describe theoverlapping regions.

Specifically, when only the front camera 2 is specified as therecognition device, the virtual viewpoint E1 is set using the basiccorrespondence table. The virtual viewpoint E1 is a predetermined anglefrom a point at the rear-side, obliquely above the host vehicle, to thefront-side obliquely towards a low part thereof, which includes the hostvehicle (refer to FIG. 5 (A)). In a case of when only the rear camera 4is specified as the recognition device, the virtual viewpoint E2 is setas a predetermined angle, from a point at the front-side, obliquelyabove the host vehicle to the rear-side obliquely towards a lower partthereof which includes the host vehicle (refer to FIG. 5 (A)).

In the same way, if the specified camera 10 is only the right-sidecamera 6, the virtual viewpoint E3 is set as a predetermined angle froma point on left-side obliquely above the host vehicle to the right sideobliquely towards a lower part thereof which includes the vehicle.Additionally, if only the left-side camera 8 is specified as therecognition device, the virtual viewpoint E4 is set as an angle from apoint on the right side, obliquely above the host vehicle to theleft-side, obliquely towards a lower part, which includes the vehicle(refer to FIG. 5 (B)).

Also setting of virtual viewpoint using the supplementary table conformsto the setting of the virtual viewpoint using the basic correspondencetable. That is, by setting of the viewpoints using the correspondencetables, specifically the basic correspondence table and thesupplementary correspondence table, the virtual viewpoints E5 to E8 areeach set as the predetermined angles from a position obliquely above therecognition camera on opposed-side thereof, obliquely towards a lowerpart of the recognition camera-side.

It is to be understood that a direction of the virtual viewpoint ispre-set as a predetermined angle within a range that does not satisfy aright angle (that is, a perpendicular direction) (for example, between 0to 80°) in at least a vehicle height direction of the host vehicle. Thereason for such angle settings is that if a viewpoint converted image isproduced as a bird's-eye view image with the virtual viewpoint providedin a perpendicular direction (that is 90°), the tendency of the objectin the image stretching in a vertical direction thereof as the objectmoves away from a center of the image is profoundly increased. Incontrast, if the direction of the virtual viewpoint is a flat direction(that is 0°), a blind spot region size caused by the host vehicle in theimage is maximized. Hence, in the first embodiment, a set angle relatedto the direction of the virtual viewpoint is within an angle range inwhich a perpendicular direction or flat direction of the host vehicle isnot met (for example, a range between 10 to 80°).

The image synthesis unit 23 is provided with a function of generating asynthesized image. The synthesized image is generated using eachviewpoint image supplied from the viewpoint conversion unit 22 joinedtogether to partially overlap. In the first embodiment, a region whichincludes a joined section in the synthesized image is an overlappingimage region. Specifically, an overlapping image region which is theviewpoint converted images of the respective front camera 2 and theright-side camera partial overlapped, and an overlapping image regionwhich is the viewpoint converted images of the respective front camera 2and the left-side camera 8 partially overlapped are each formed.Additionally, an overlapping image region which is the viewpointconverted images of the respective rear camera 4 and the right-sidecamera 6 partially overlapped, and an overlapping region of theviewpoint converted images of the respective rear camera 4 and theleft-side camera 8 partially overlapping are each formed. Theoverlapping regions described above are thus the regions which includethe joined sections of the viewpoint converted images in the synthesizedimage. It is to be understood that each overlapping region correspondsto the overlapping regions OA1 to OA4 (refer to FIG. 2) of the imagingregions of each camera 10.

The image synthesis unit 23 is provided with the function ofsynthesizing each viewpoint converted image of the overlapping imageregion using a preset blend ratio (percentage), when the synthesizedimage is generated. The reason for this is by synthesizing eachviewpoint converted image of the overlapping image region included inthe joined section of the synthesized image, the joined sections becomeless obvious in the synthesized image, hence, unnatural appearance, forexample, distortion of the joined sections can be reduced.

It is noted that, in the first embodiment, since a synthesized image isformed having the entire viewpoint images of each camera 10 joined bypartial overlapping, an around view image with decrease distortion maybe produced. The around view image, as shown in FIG. 6 (A) is an imagewhich can display an entire surrounding area of the host vehicle inaddition to the host vehicle itself.

The image-setting unit 24 is provided with a function of setting theblend ratio (percentage) of each viewpoint converted image (alsoreferred to as an overlapping image region area hereon) of theoverlapping image regions, on the basis of recognition result of anobject, recognized by the image recognition unit. The blend ratio set bythe image-setting unit 24 is used for the synthesis of the overlappingimage region areas performed by the image synthesis unit 23. It is to beunderstood that a process, which actualizes a function of theimage-setting unit 24 (referred to as a blend ratio setting processhereon) will be described in detail hereinafter.

The image enhancement unit 25 is provided with function of visuallyenhancing an image area of an object (referred to as an object imagearea hereon) included in the viewpoint converted image converted by theviewpoint conversion unit 22. Specifically, in the first embodiment, aprocess to visually enhance the object image area of the synthesizedimage generated at the image synthesis unit 23 is performed. The processis actualized by specifying an image position of the object, on thebasis of the image position information stored in the memory 14.Regarding the visual enhancement of an object image area, an imagesurrounding an object, or an enhanced image which visually enhances anobject in the image, for example may be added to the synthesized image.A process in which brightness of the object image area may be increasedto be higher than other surrounding image areas, or a contrast of atleast one of the object image area and the synthesized image may also bechanged.

The image-outputting unit 26 is provided with a function of outputting asynthesized image, which is synthesized by the overlapping image regionareas using the image synthesis unit 23. The synthesized image has ablend ratio set at the image-setting unit 24 and is output to thedisplay 30 as the driving support image. Specifically, according to thefirst embodiment, the around view image having the object image areavisually enhanced by the image enhancement unit 25 is output to thedisplay 30. The driving support image is a display image which notifiesthe driver of an existing object, for example, another vehicle or apedestrian, hence the driving support image is a displayed image whichassists driving of the host vehicle.

[Process] [Virtual Viewpoint Setting Process]

Next a virtual viewpoint setting process executed by the CPU 12 in orderto actualize a partial function of the viewpoint conversion unit 22 isdescribed using a flowchart shown in FIG. 7. It is to be understood thatthe process is repeatedly performed at a predetermined timing for eachfunction inside the display control unit 20, during a period in which aswitch (not shown) of the display control apparatus 1 mounted in avehicle is switched on.

Once the process starts, firstly, at the viewpoint conversion unit 22,captured images from all of the cameras 10 are input, and determinationof whether a captured image has an object is performed by the imagerecognition unit step S110. The determination is performed, for example,based on whether identification information is stored in the memory 14.If it is determined that a captured image which includes an object inthe image exists (among the captured images), the process proceeds tostep S120, and if it is determined that such an image does not exist,the process proceeds to step S150.

At step S120, a process of selecting either the basic correspondencetable or the supplementary correspondence table used to set the virtualviewpoint is performed (the process is referred to as a simplified tableselection process hereon) and the process proceeds to step S130. It isnoted that the recognition camera is also specified when either thebasic correspondence table or the supplementary correspondence table isselected at the simplified table selection process. The simplified tableselection process will be described in detailed hereinafter.

At step S130, it is determined whether selection of either the basiccorrespondence table or the supplementary correspondence table has beenperformed at step S120. The selection mentioned here of which isperformed using the simplified table selection process. At step S130, ifit is determined that either one of the correspondence tables areselected, the process proceeds to step S140, in contrast if neither ofthe correspondence tables are selected the process proceeds to stepS150.

At step S140, the virtual viewpoint is set using the correspondencetable selected at step S120 by the simplified table selection process,and the process is completed. It is noted that in the first embodiment,the virtual viewpoint is set as a predetermined angle from a positionobliquely above the recognition camera on opposed-side thereof, andobliquely towards a lower part of the recognition camera-side.

In contrast, at step S150, a camera having the highest priority level(referred to as ‘priority camera’ hereon) is selected by a method, whichis different from the simplified table process of step S120. A processwhich specifies the selected priority camera as the recognition camera(referred to hereon as ‘a recognition camera priority process’) isperformed, and the process proceeds to step S160. Specifically, apriority camera is selected using a different method for each reason therecognized camera could not be recognized.

That is, in the present embodiment, in a case of determining that noobject exists in the images at step S110, for example, one to twocameras are selected as priority cameras. It is noted that, a drivingdirection of the host vehicle is specified based on vehicle informationtransmitted through the in-vehicle LAN from other ECUs, for example. Ina another example, if the number of recognition cameras are 3 or more,or if there are two recognition cameras and imaging region conditionsdescribed hereinafter are not satisfied, a priority camera 10 such asdescribed below is selected as the priority camera. That is, thepriority camera in this case is the camera 10 which captures an imagehaving the highest number of objects, the camera 10 which capturesimages including an object in an imaging region corresponding to thedriving direction of the host vehicle, and the camera 10 which isadjacent to two of three cameras 10 having a captured image with anobject included in the image, for example. The priority camera isspecified on the basis of the identification information and the imageposition information stored in the memory 14.

At step S160, either one of the corresponding tables, among the basiccorrespondence table and the supplementary correspondence table isselected, according to the priority camera specified at step S150 by therecognition camera priority process. Thereafter, the virtual viewpointis set using the selected correspondence table and the process ends.That is, in the first embodiment, since the specified priority camera isthe recognition camera, a virtual viewpoint is set as the predeterminedangle from a position obliquely above the recognition camera on anopposed-side thereof, and obliquely towards a lower part of therecognition camera-side.

[Simplified Table Selection Process]

Next, the simplified table process executed by the CPU12 in step S120 isdescribed using the flow chart shown in FIG. 8.

At step S210, once the process is initiated, firstly, the recognitioncamera is specified at the viewpoint conversion unit 22 based on theidentification information stored in memory 14, and it is determinedwhether the specified camera is only one camera or more than one camera.If one recognition camera is determined, the process proceeds to stepS220, the basic correspondence table is selected and the process ends.In contrast, if the recognition camera is determined as not being only 1camera, (in the first embodiment the recognition camera is two or more),the process proceeds to step S230 and it is determined whether thenumber of specified cameras are 2 cameras. Furthermore, if the number ofspecified cameras is determined as two cameras, the process proceeds tostep S240, however when the specified camera is determined as 3 or morecameras, the process ends.

An imaged region, which includes an object recognized by the imagerecognition unit 21, is an object-imaged region. At step S240, it isdetermined whether predetermined image region conditions of theobject-imaged region are satisfied. If it is determined that the imagingregion conditions are satisfied, the process proceeds to step S250, thesupplementary correspondence table is selected and the process ends. Ifit is determined that the imaging region conditions are not satisfied,the process ends without selection of a correspondence table.Specifically, in the first embodiment, the object-imaged region isspecified based on the image position information stored in the memory14.

For example, one of the overlapping regions among the overlappingregions OA1 to OA4 may be an object-imaged region condition. That is, ifan object recognized by the image recognition unit 21 exists in one ofthe overlapping regions OA1 to OA4 even in a case of two recognitioncameras being specified, a virtual viewpoint which corresponding to thetwo recognition cameras combined may be set. This may be performed byselection of the supplementary table since both recognition camerascapture the object.

In another example, two of the single regions SA1 to SA4 both adjacentto one of the overlapping regions OA1 to OA4, may be also satisfy animaging region condition. That is, if two or more objects are recognizedto be distributed in two of the single regions SA1 to SA4, sandwichedbetween one of the overlapping regions OA1 to OA4, which are differentimaging regions, a virtual viewpoint corresponding to two of the camerascombined may be set.

It is noted that the imaging region conditions are not limited to theexemplary embodiment, and a plurality of conditions may bepredetermined.

[Blend Ratio Setting Process]

Next, a virtual viewpoint setting process performed by the CPU12 toactualize the function of the image-setting unit 24 will be describedwith reference to the flow chart shown in FIG. 9. It is noted that theprocess, for example, may be repeatedly initiated at predeterminedtimings for every function of the display control unit 20, during a timein which a switch of the display control apparatus 1 is switched on.

Once the process starts, firstly at step S310, it is determined whetherthe object recognized by the image recognition unit 21 exists in one ofthe overlapping regions OA1 to OA4, at the image-setting unit 24. Thedetermination may be performed on the basis of the image positioninformation stored in the memory 14, for example. When it is determinedthat an object exists in one of the overlapping areas OA1 to OA4, theprocedure proceeds to step S320.

In contrast, when it is determined that an object does not exist on theoverlapping regions OA1 to OA4, the process proceeds to step S360, aninitial setting of 50% blend ratio is continually set for each viewpointconverted image area for the entire overlapping image regions and theprocess ends. The blend ratio is a synthesizing percentage of a pixelvalue (for example, RGB value) of each viewpoint converted image area inthe overlapped imaging regions. As a result, if the blend ratio is setat 50% for each viewpoint converted image, and each viewpoint convertedimage area is given as the respective image area B1 and image area C1before synthesizing the overlapping image regions, an image area B2 ofeach pixel value configuring the image area B1 multiplied by 50% and animage area C2 of each pixel value configuring the image area C1multiplied by 50% are respectively added.

At step S320, the recognition results of an object determined to existare acquired, and the process proceeds to step S330. Specifically, inthe present embodiment, the recognition results of an object may beobtained by reading a candidate value and optical flow vector value, forexample, related to the an object stored in the memory 14.

At step S330, a region which is determined at step S310 to have anobject existing among the overlapping regions OA1 to OA4 is defined asan object overlapping region. A recognition precision of an object ineach captured image of the two cameras 10, which captures images inthese regions are compared. Specifically, the recognition precision ofan object in each of the object overlapping regions, captured by the twocameras 10 is compared. At step S330, it is determined whether therecognition precision of an object is different in both captured images.Specifically, in the present embodiment, it may be determined that anobject with a larger candidate value has a higher recognition precision,among the recognition result acquired at step S320.

It is noted that step S320 is a recognition results acquiring meanswhich acquires the recognition result functionally using a processexecuted using the CPU 12.

In this way, if it is determined that the recognition precision ofobjects between each captured image is varying, the procedure proceedsto step S370. At step S370, the blend ratio of the overlapping imageregion in which an object is distributed is adjustably set, and theprocess ends. Specifically, the of a viewpoint converted image areabased on a captured image having a the highest recognition precision isset higher than the blend ratio of other viewpoint converted imageareas, among the viewpoint converted images in the overlapping regions.For example, in the present embodiment, the higher the recognitionprecision is the larger the blend ratio may be set, according to therecognition precision of each object in both captured images areas.

That is, in the settings described, it is necessary to set the blendratio so that the blend ratios of both captured images when addedtogether is 100%. For example, the image area B2 having each pixel valuewhich configures the image area B1 multiplied by 70%, and the image areaC2 having each pixel value which configures the image area C1 multipliedby 30% can be added together. In this case, the image area B1 is theimaging area which has the highest recognition precision, and the imagearea C1 is the imaging area which has the lowest recognition precision,among each viewpoint converted image area in the overlapping areas.

In contrast, when it is determined that the recognition precision of anobject is not different between each of the captured images, theprocedure proceeds to step S340. At step S340, a warning priority levelof an object is compared in each captured image of the two cameras 10which capture these regions, and it is determined whether the warningpriority level of the objects in both images are different, for eachobject overlapping region. Specifically, in the present embodiment, itmay be determined that, the higher the optical flow value related to anobject is the higher the warning priority level is, among the recognizedresults acquired at step S320. It is noted that, comparison of thewarning priority levels is not limited to the method described above.That is, the warning priority level may be determined, for example, byan object type in the image and other indicators.

The step S340 described above is a warning priority level determinationmeans which determines the priority level of an object by function of aprocess executed by the CPU12.

In this manner, even when it is determined that the warning prioritylevel of an object between images is different, the procedure proceedsto step S370, the blend ratio of the overlapping image region area inwhich the object is positioned in the image is adjusted and the processends. Specifically, the of the viewpoint converted image area is sethigher than the of other viewpoint converted images, based on ancaptured image which has the highest warning priority level among theeach of the viewpoint converted images in the overlapping image regions.For example, in the present embodiment, the higher the warning prioritylevel is the higher the blend ratio may be set, in accordance to eachwarning priority level of an object in both captured image areas.

In contrast, when it is determined that the warning priority level of anobject between each captured image is not different, the processproceeds to step S350 and the image position of the object is comparedbetween each captured image area of the two cameras 10 designated toimaging regions of the captured images. At step S350, it is determinedwhether one of the captured image areas, among the two captured imageareas, satisfies the predetermined object position conditions, relatedto the position of the object. Specifically, in the present embodiment,for the position of the object in the image, a distance from a center ofimage is compared, based on the image position information stored in thememory 14. For example, it may be determined that the distance issmaller than a predetermined threshold, as a requirement to fulfill theobject position conditions.

The object position conditions are not limited to the examples describedabove, that is, conditions related to an actual position of an objectand a position of an object in the image may also be prescribed.

Implementation of conditions, which can directly determine the positionof the object base on a position in the image, can decrease the load ofperforming a process.

In this manner, even when the existence of a captured image area whichsatisfies the object position condition is determined, among each of thecaptured images, the process proceeds to step S370, and the blend ratioof the overlapping image region areas in which the object is positionedis adjustably set and the process ends. Specifically, a viewpointconverted image area that is based on an image having satisfied objectposition conditions is set to have a higher blend ratio than the otherviewpoint converted image areas, among each of the viewpoint convertedimages of the overlapping image regions. For example, in the firstembodiment, the shorter the distance of the object from the center ofeach captured image is, the higher the blend ratio may be set, accordingto the image position of the object in both captured images.

When it is determined that captured image areas which satisfy the objectposition condition does not exist between each of the captured images,the procedure continues to step S360, the initial setting in which theblend ratio (%) of each viewpoint converted image is set to 50% iscontinued for all of the overlapping image regions, and the processends.

In this manner, the blend ratio of each viewpoint converted image areaof the overlapping image region is set according to the recognitionprecision, the warning priority level, and the object positionconditions in the process described above. FIG. 10 shows a situation inwhich a pedestrian is captured as the object by the rear camera 4 andthe right-side camera 6. That is, FIG. 10 shows a situation in which therear camera 4 (refer to FIG. 10 (A)) has a higher recognition precisionof both captured images, than the right-side camera 6 (FIG. 10B)). Withreference to FIG. 10 (B), in this example, if the blend ratio (%) is setto 50% for each viewpoint converted image area of the overlapping imageregions, the driving support images easily deteriorates visually, as thepedestrian is synthesized and shown at the same blend ratio (%) in bothviewpoint converted images. In contrast, if the blend ratio is set to70% for the rear camera 4, which has high recognition precision, and setto 30% for the right-side camera 6, which has low recognition precision,the visual deterioration of the driving support image is suppressed, asthe image with the high recognition precision is synthesized so that thepedestrian stands out in the image (refer to FIG. 10(D)).

[Effect]

The following effects may be obtained from the first embodiment. It isnoted that, the viewpoint converted image and the viewpoint convertedimage area will be respectively referred to as a converted image and aconverted image area, for simplicity.

Each converted image area in an overlapping image region is synthesizedby partial overlapping to join each converted image area. Furthermore,as each converted image area is synthesized with a set blend ratio basedon the recognition results of an object, enhancement of the joinedsection is omitted, and an object may be clearly presented in theoverlapping image region area which includes the joined section. As afurther result, unnatural appearance, such as distortion of the joinedsections may be resolved by synthesizing the overlapping image regionareas of each converted image, and distortion of the visibility of anobject in the overlapping image region areas may be decreased.

In the blend ratio setting process, among each of the converted imageareas in the overlapping image regions, the blend ratio of a convertedimage is set to be higher than a blend ratio of other converted imageareas, based on a captured image with the highest recognition precisionof an object. As a result, an object with the highest warning prioritylevel may be displayed to stand out, in the driving support image.

Also in the blend ratio setting process, among each of the convertedimage areas in the overlapping image regions, the blend ratio of aconverted image area is set to be higher than the blend ratio of otherconverted image areas, based on a captured image including an objectwith the highest warning priority. As a result, the driving supportimage may be provided so that an object which has the highest warningpriority may be presented to stand out in the driving support image.

The blend ratio (%) of the converted image area is set higher than theblend ratio of other converted image areas, for a captured image inwhich the predetermined object position conditions are satisfied, amongeach converted image area in the overlapping image regions. In thiscase, the predetermined object conditions specifically relate a positionof the object. As a further result, the driving support image may bepresented so that an object which has the highest possibility of beingcompletely captured is provided to visually stand out in an image.

Other Embodiments

A preferred embodiment of the present disclosure has been describedherein above, however the present disclosure is not limited to thepreferred embodiment and other various modes may be adapted.

In the preferred embodiment, the captured images of each camera 10synthesized viewpoint converted images with converted viewpoints, andthe synthesized images are output as the driving support image, howeverthe driving support image is not limited to the above-described process.For example, at least one captured image from each camera 10 may beconverted to a viewpoint converted image and output as the drivingsupport image.

A configuring element of the preferred embodiment having a plurality offunctions may be actualized by a plurality of elements, and a pluralityof configuring elements provided with one function may be unified intoone element. Some of the configuring elements of the preferredembodiment may be omitted, and at least some of the configuring elementsof the preferred embodiment may be added to the other embodiments orsubstituted by a different element. It is to be understood that allmodes included in the technical idea specified by the scope of theclaims are the embodiments of the present disclosure.

In addition to the display control apparatus 1 installed in a vehicle, asystem configuring the display control apparatus 1, one or more programsto configure a computer to function as the display control apparatus 1,one or more recording medias (specifically, a non-transitory recordingmedia, for example, a semiconductor memory), and a display controllingmethod, for example, according to the present disclosure, may beactualized by a variety of modes.

SYMBOLS

1 . . . display control apparatus installed in a vehicle

2 . . . front camera

4 . . . rear camera

6 . . . right-side camera

8 . . . left-side camera

10 . . . camera

12 . . . CPU

14 . . . memory

20 . . . display control unit

21 . . . image recognition unit

22 . . . viewpoint conversion unit

23 . . . image synthesis unit

24 . . . image-setting unit

25 . . . image enhancing unit

26 . . . image-outputting unit

30 . . . display

A1 . . . front region

A2 . . . rear region

A3 . . . right-side region

A4 . . . left-side region

E1 to E8 . . . virtual viewpoints

OA1 . . . front right-side overlapping region

OA2 . . . front left-side overlapping region

OA3 . . . rear right-side overlapping region

OA4 . . . rear left-side overlapping region

SA1 . . . front single region

SA2 . . . rear single region

SA3 . . . right-side single region

SA4 . . . left-side single region

1. A display control apparatus mounted in a vehicle, the apparatuscomprising: a plurality of imaging devices provided on the vehicle tocapture a preset plurality of imaging regions which overlap in part,surrounding the vehicle, by dividing the imaging regions among therespective imaging devices; an image recognition unit which recognizes apredetermined object in images captured by of each of the plurality ofimaging devices; an image synthesis unit which synthesizes an image to aconverted image viewed from a virtual viewpoint which is preset, foreach image captured by two imaging devices when the imaging regionswhich include the object recognized by the image recognition unit areoverlapping imaging regions in which both of two imaging devicescaptures images; an image-setting unit which sets a blend ratio of eachconverted image area in the overlapping image regions based on arecognition result of the object recognized by the image recognitionunit, and an image-outputting unit which outputs the synthesized imageas a driving support image to a display mounted in a vehicle, thesynthesized image being synthesized from each converted image area inthe overlapping image regions, using the image synthesis unit, and theblend ratio of the each converted image being preset by theimage-setting unit, wherein, the image synthesis unit synthesizes eachconverted image area in the overlapping image regions of two convertedimages overlapping, using a preset blend ratio, to generate asynthesized image by overlapping to join each converted image, and theimage-setting unit is configured to set the blend ratio of a convertedimage area higher than the blend ratio of other conversion image areas,for a captured image in which predetermined object position conditionswhich relate to an object position are satisfied, among each convertedimage area in the overlapping image regions, a distance of the objectposition from a center of each captured image of the two imaging devicesbeing compared, and the shorter the distance of the object is from thecenter of the image, the higher the blend ratio of the image is set,when the blend ratio of the converted image is set higher than the blendratio of other conversion image areas.
 2. The display control apparatusaccording to claim 1, wherein, the image-setting unit is provided with:a setting means for setting the blend ratio of a converted image areahigher than the blend ratio of other converted image areas, for acaptured image which has a highest recognition precision of the object,among each converted image area in the overlapping image regions.
 3. Thedisplay control apparatus according to claim 1, wherein, theimage-setting unit is provided with a warning priority leveldetermination means for determining whether the warning priority levelchanges between images, and the setting means for setting the blendratio of a converted image area higher than the blend ratio of otherconverted image areas, for a captured image which has a highest warningpriority level of the object, among each converted image area in theoverlapping areas.
 4. (canceled)
 5. A display control apparatus methodfor a vehicle, the method comprising: an image recognition step forrecognizing predetermined objects in captured images captured by aplurality of imaging devices mounted on the vehicle for capturing eachimaging region which overlap in part surrounding the vehicle; an imagesynthesis step for synthesizing an image to a converted image viewedfrom a virtual viewpoint which is preset, for each image captured by twoimaging devices when the imaging regions which include the objectrecognized by the image recognition unit are overlapping imaging regionsin which both of two imaging devices captures images; an image settingstep for setting a blend ratio of each converted image area in theoverlapping image regions based on a recognition result of the objectrecognized by the image recognition unit, and an image outputting stepfor outputting the synthesized image as a driving support image to adisplay mounted in a vehicle, the synthesized image being synthesizedusing each converted image area in the overlapping image regions at theimage synthesis step and the blend ratio of the each converted imagebeing set by the image setting step, wherein, the image synthesis stepsynthesizes each converted image area in the overlapping image regionsof two converted images overlapping, using a preset blend ratio, togenerate a synthesized image by overlapping to join each convertedimage, and the image-setting step is performed to set the blend ratio ofa converted image area higher than the blend ratio of other conversionimage areas, for a captured image in which predetermined object positionconditions which relate to an object position are satisfied, among eachconverted image area in the overlapping image regions, a distance of theobject position from a center of each captured image of the two imagingdevices being compared, and the shorter the distance of the object isfrom the center of the image, the higher the blend ratio of the image isset, when the blend ratio of the converted image is set higher than theblend ratio of other conversion image areas.
 6. A processing device, theprocessing device comprising: an image recognition unit which recognizespredetermined objects in captured images captured by a plurality ofimaging devices mounted on a vehicle for imaging a plurality ofpredetermined imaging areas, which overlap in part surrounding thevehicle, by dividing the imaging regions among the imaging devices; animage synthesis unit which synthesizes an image to a converted imageviewed from a virtual viewpoint which is preset, for each image capturedby two imaging devices, when the imaging regions which include theobject recognized by the image recognition unit are overlapping imagingregions in which both of two imaging devices captures images; animage-setting unit which sets a blend ratio of each converted image areain the overlapping image regions based on a recognition result of theobject recognized by the image recognition unit, and an image-outputtingunit which outputs the synthesized image as a driving support image to adisplay mounted in a vehicle, the synthesized image being synthesizedfrom each converted image area in the overlapping image regions, usingthe image synthesis unit, and the blend ratio of the each convertedimage being preset by the image-setting unit, wherein, the imagesynthesis unit synthesizes each converted image area in the overlappingimage regions of two converted images overlapping, using a preset blendratio, to generate a synthesized image by overlapping to join eachconverted image, and the image-setting unit is configured to set theblend ratio of a converted image area higher than the blend ratio ofother conversion image areas, for a captured image in whichpredetermined object position conditions which relate to an objectposition are satisfied, among each converted image area in theoverlapping image regions, a distance of the object position from acenter of each captured image of the two imaging devices being compared,and the shorter the distance of the object is from the center of theimage, the higher the blend ratio of the image is set, when the blendratio of the converted image is set higher than the blend ratio of otherconversion image areas.
 7. The processing device according to claim 6,wherein, the image-setting unit is provided with a setting means forsetting the blend ratio of a converted image area higher than the blendratio of other converted areas for an image having on a highestrecognition precision of the object, recognized by the recognitionresult acquiring means, among each of the converted image areas in theoverlapping image regions.
 8. The processing device according to claim6, wherein; the image-setting unit is provided with a warning prioritylevel determination means for determining whether the warning prioritylevel between images changes, and the setting means for setting theblend ratio of a converted image area higher than the blend ratio ofother converted areas for a captured image which is determined to have ahighest warning priority level of the object, determined by the warningpriority level determination means, among the each converted image inthe overlapping regions.
 9. (canceled)
 10. A recording medium in whichdigital program data is stored, the program data being readably by aCPU, the CPU reading the program data from the recoding medium andexecuting the read program data so as to enable the CPU to function as:an imaging recognition unit which recognizes predetermined objects incaptured images captured by a plurality of imaging devices mounted on avehicle for imaging each imaging region which overlaps in part,surrounding the vehicle, by dividing the imaging regions among theimaging devices; an image synthesis unit which synthesizes an image to aconverted image viewed from a virtual viewpoint which is preset, foreach image captured by two imaging devices, when the imaging regionswhich include the object recognized by the image recognition unit areoverlapping imaging regions in which both of two imaging devicescaptures images; an image-setting unit which sets a blend ratio of eachconverted image area in the overlapping image regions based on arecognition result of the object recognized by the image recognitionunit, and an image-outputting unit which outputs the synthesized imageas a driving support image to a display mounted in a vehicle, thesynthesized image being synthesized by each converted image area in theoverlapping image regions, using the image synthesis unit, and the blendratio of the each converted image being preset by the image-settingunit, wherein, the image synthesis unit synthesizes each converted imagearea in the overlapping image regions of two converted imagesoverlapping, using a preset blend ratio, to generate a synthesized imageby overlapping to join each converted image, and the image-setting unitis configured to set the blend ratio of a converted image area higherthan the blend ratio of other conversion image areas, for a capturedimage in which predetermined object position conditions which relate toan object position are satisfied, among each converted image area in theoverlapping image regions, a distance of the object position from acenter of each captured image of the two imaging devices being compared,and the shorter the distance of the object is from the center of theimage, the higher the blend ratio of the image is set, when the blendratio of the converted image is set higher than the blend ratio of otherconversion image areas.
 11. The display control apparatus according toclaim 2, wherein, the image-setting unit is provided with a warningpriority level determination means for determining whether the warningpriority level changes between images, and the first setting means forsetting the blend ratio of a converted image area higher than the blendratio of other converted image areas, for a captured image which has ahighest warning priority level of the object, among each converted imagearea in the overlapping areas.
 12. The processing device according toclaim 7, wherein; the image-setting unit is provided with a warningpriority level determination means for determining whether the warningpriority level between images changes, and the first setting means forsetting the blend ratio of a converted image area higher than the blendratio of other converted areas for a captured image which is determinedto have a highest warning priority level of the object, determined bythe warning priority level determination means, among the each convertedimage in the overlapping regions.